Subject tracking apparatus, control method, image processing apparatus, and image pickup apparatus

ABSTRACT

The subject tracking apparatus comprises: a first registering unit configured to register a partial area as a template indicative of a subject in one image of supplied images; a first matching unit configured to estimate a subject area by collating a partial area in newly supplied images with the template registered by the first registering unit; a second registering unit configured to register a histogram generated based on a pixel value of a partial area indicative of the subject in one image of supplied images; a second matching unit configured to estimate a subject area by collating a histogram of a partial area in newly supplied images with the histogram registered by the second registering unit; and a tracking area determination unit configured to determine a tracking area based on estimation results by the first matching unit and the second matching unit. The first registering unit and the second registering unit allow at least one of the template and the histogram previously registered to be updated, and the update for the registration of the template by the first registering unit is more frequently than that for the registration of the histogram by the second registering unit.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to a subject tracking apparatus, a controlmethod, an image processing apparatus, and an image pickup apparatus,and more specifically, a subject tracking apparatus for tracking asubject included in images which are sequentially supplied.

Description of the Related Art

In recent years, a technique in which a particular subject is extractedfrom images supplied sequentially in a time series manner and then, theextracted subject is tracked has been very useful. For example, thetechnique has been used for specifying a human face region and a humanbody region in dynamic images. Such a technique can be used in a numberof different fields, such as teleconferences, man-machine interfaces,security, monitoring systems for tracking any subjects, and imagecompression.

Also, in digital still cameras, digital video cameras and the like,there is a well-known technique for extracting and tracking any subjectincluded in a photographing image, and thereby optimizing a focal pointstate and an exposure state with respect to the subject (see, JapanesePatent Laid-Open No. 2005-318554, Japanese Patent Laid-Open No.2001-060269, and Japanese Patent Laid-Open No. 2004-348273).

For example, Japanese Patent Laid-Open No. 2005-318554 discloses animage pickup apparatus for detecting (extracting) and tracking, withrespect to the face, a position of a face in a photographed image andimaging the face at optical exposure while fitting the face to a focalpoint. Tracking the detected face enables stable control in a timeseries manner.

Also, Japanese Patent Laid-Open No. 2001-060269 discloses a techniquesuch that a certain subject is automatically tracked by utilizing atemplate matching. The template matching is a technique for registeringa partial image as a template image by clipping an image area with acertain subject set as a target to be tracked, and estimating an areawith the highest degree of similarity, or the least degree of differencerelative to the template image, and tracking the certain subject.

Japanese Patent Laid-Open No. 2004-348273 discloses, in contrast to thetemplate matching, a technique, referred to as a histogram matching, forutilizing a histogram with respect to the amount of characteristic forthe matching, rather than image data itself. This is a technique forconverting information indicative of a subject from image data into ahistogram, registering the histogram, estimating an area capable ofbeing converted into the histogram most similar to the registeredhistogram among the images, and tracking the certain subject.

However, while template matching is good at classification betweensubjects that are similar to each other, it is weak at classificationwhen there are changes in the appearance because template matching usespatterns of the image data as the amount of characteristic. In contrast,histogram matching can realize robust tracking of the subject withrespect to changes in the appearance, such as a change in the attitudeof the subject, by converting information indicative of the subject fromthe image data into a histogram for the sake of ambiguity. However, itis weak at the classification between the similar subjects. In addition,it is difficult to use histogram matching simply in combination with thetemplate matching because the properties thereof are different from eachother.

SUMMARY OF THE INVENTION

The present invention provides a subject tracking apparatus capable ofimproving performance for tracking a subject, while combining histogrammatching with template matching.

The subject tracking apparatus according to the present invention is asubject tracking apparatus that tracks a subject included in images thatare sequentially supplied, the apparatus comprising: a first registeringunit configured to register a partial area indicative of the subject inone image of the supplied images as a template; a first matching unitconfigured to estimate a subject area by collating a partial area in anewly supplied image with the template registered by the firstregistering unit; a second registering unit configured to generate ahistogram based on a pixel value of a partial area indicative of thesubject in one image of the supplied images and register the generatedhistogram; a second matching unit configured to estimate a subject areaby collating a histogram of a partial area in a newly supplied imagewith the histogram registered by the second registering unit; and atracking area determination unit configured to determine a tracking areabased on estimation results by the first matching unit and the secondmatching unit; wherein the first registering unit and the secondregistering unit are allowed to update at least one of the template andthe histogram previously registered; and wherein the update for theregistration of the template by the first registering unit is morefrequent than that for the registration of the histogram by the secondregistering unit.

According to the present invention, there can be provided the subjecttracking apparatus capable of estimating the subject area with highprecision and improving the performance for tracking the subject, whilecombining template matching with histogram matching.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a schematic configuration of animage pickup apparatus.

FIG. 2 is a block diagram illustrating a configuration of a subjecttracking circuit.

FIG. 3A and FIG. 3B are diagrams for illustrating a template matching.

FIG. 4A and FIG. 4B are diagrams for illustrating a histogram matching.

FIG. 5 is a diagram illustrating a framework of a subject tracking.

FIG. 6 is a flowchart illustrating a subject tracking process.

FIG. 7 is a flowchart illustrating subject area determinationprocessing.

FIG. 8 is a flowchart illustrating a flow for optimized processing in ahistogram acquiring area.

DESCRIPTION OF THE EMBODIMENTS

Hereinafter, a description will be given of each preferred embodiment ofthe present invention with reference to the accompanying drawings andthe like. The present invention can be applied to an image pickupapparatus such as a digital still camera, and a digital video camera.

FIG. 1 is a block diagram illustrating a schematic configuration of animage pickup apparatus 100 according to an embodiment of the presentinvention. In the present embodiment, the image pickup apparatus 100 isembodied as a digital camera for capturing an image of a subject. Also,the image pickup apparatus 100 functions as a subject tracking apparatusfor tracking the subject included in images supplied (input)sequentially in a time series manner.

The image pickup apparatus 100 comprises an imaging optical system 101,an imaging element 102, an analog signal processing circuit 103, an A/Dconverter 104, a control circuit 105, an image processing circuit 106, adisplay device 107, a record medium 108, and a subject tracking circuit109.

Light indicative of the image of the subject is converged by the imagingoptical system 101, and the converged light is incident to the imagingelement 102 configured by a CCD image sensor, a CMOS image sensor, orthe like. The imaging element 102 outputs an electric signal in pixelunits, in accordance with the intensity of the incident light. In otherwords, the image of the subject formed by the imaging optical system 101is photoelectrically converted. The electric signal output from theimaging element 102 is an analog image signal indicative of the image ofthe subject captured in the imaging element 102.

With respect to an image signal output from the imaging element 102,analog signal processing such as correlated double sampling (CDS) isperformed by the analog signal processing circuit 103. The image signaloutput from the analog signal processing circuit 103 is converted into adigital data format by the A/D converter 104, and then input to thecontrol circuit 105 and the image processing circuit 106.

The control circuit 105 is configured by a CPU, a microcontroller, andthe like, which centrally controls the operation of the image pickupapparatus 100. The control circuit 105 controls imaging conditions suchas a focal point state and an exposure state when the imaging isperformed in the imaging element 102. More specifically, the controlcircuit 105 controls a focus controlling mechanism and an exposurecontrolling mechanism (neither of which is shown) of the imaging opticalsystem 101, based on the image signal output from the A/D converter 104.For example, the focus controlling mechanism is an actuator and the likeconfigured to allow a lens included in the imaging optical system 101 tobe driven in an optical axis direction, while the exposure controllingmechanism is an actuator and the like configured to allow an aperture ora shutter to be driven. Also, the control circuit 105 controls readingof the imaging element 102, such as the output timing and the outputpixels of the imaging element 102. The control circuit 105 deploysprogram codes stored in a ROM (Read Only Memory) in a work area of a RAM(Random Access Memory) and sequentially executes the deployed programcodes, thereby controlling each unit of the image pickup apparatus 100.

The image processing circuit 106 performs image processing such as gammacorrection, white balance processing, and the like, with respect to theimage signal output from the A/D converter 104. Also, in addition to thenormal image processing, the image processing circuit 106 has a functionfor performing image processing with information about the subject areain the image supplied from the subject tracking circuit 109 as describedbelow.

The image signal output from the image processing circuit 106 istransmitted to the display device 107. The display device 107 isconfigured, for example, by an LCD, or an organic EL display, anddisplays the image signal. The images captured sequentially in a timeseries manner in the imaging element 102 are sequentially displayed inthe display device 107, thereby allowing the display unit 107 tofunction as an electronic viewfinder (EVF). Also, the display device 107displays the subject area including the subject tracked by the subjecttracking circuit 109 as a rectangular shape and the like.

Also, the image signal output from the image processing circuit 106 isrecorded on the record medium 108 (for example, a removable memory card,and the like). It is to be noted that the recording destination of theimage signal may be a memory built in the image pickup apparatus 100 oran external device communicably connected via a communication interface(not shown).

The subject tracking circuit 109 tracks the subject included in theimages (image signals) supplied sequentially in a time series manner (inother words, each of which has a different imaging time) from the imageprocessing circuit 106. The subject tracking circuit 109 estimates thesubject area from the sequentially supplied images, based on the pixelpattern or the histogram for the subject. The detail description of thesubject tracking circuit 109 is described as below.

The control circuit 105 can use the information about the subject areasupplied from the subject tracking circuit 109 in the control of thefocus controlling mechanism and the exposure controlling mechanism asdescribed above. More specifically, the focal point control using thecontrast value of the subject area, or the exposure control using theluminance value of the subject area is performed. Thereby, in the imagepickup apparatus 100, the imaging processing can be performed, takinginto account the certain subject area in the photographed image.

Here, a detailed description will be given of the subject trackingcircuit 109. The subject tracking circuit 109 functions as two types ofmatching unit. One is a matching unit configured to estimate the areawith the high degree of similarity or the low degree of difference bycollating the partial area of the supplied images with a partial imageas the template indicative of the subject as a target and altering thecollated partial area (hereinafter, referred to as “template matching”).The other is a matching unit configured to estimate the area with thehigh degree of similarity or the low degree of difference by collatingthe histogram of the partial area of the supplied images with thehistogram of the partial area indicative of the subject as a target, andaltering the collated partial area (hereinafter, referred to as“histogram matching”). Additionally, the subject area is determinedbased on a matching evaluation value in each estimation result. In thesubject tracking according to an embodiment of the present invention,the tracking with high precision is performed by altering the area inwhich the amount of a characteristic for the tracking is acquired, andthe update frequency, in accordance with the each property of thematching units.

FIG. 2 is a block diagram of the subject tracking circuit 109. Thesubject tracking circuit 109 is comprised of a subject detecting circuit201, a template registering circuit 202, a template matching circuit203, a histogram registering circuit 204, a histogram matching circuit205, and a tracking process controlling circuit 206. Each block(circuit) from the subject detecting circuit 201 to the tracking processcontrolling circuit 206 is connected by a bus, thereby allowing eachblock to exchange the data.

The subject detecting circuit (subject detecting unit) 201 detects apredetermined subject as a target from the supplied images, andspecifies the subject as the target to be tracked. A face of a person,for example, is representative of a target subject. In this case, thesubject detecting circuit 201 specifies the face area of the person asthe subject area, and sets the face area of the person as the target tobe tracked. The detecting unit for the subject in the subject detectingcircuit 201 may use a well-known detecting unit if the subject as atarget to be detected is the face of the person. For example, thewell-known techniques for detecting the face include a method forutilizing knowledge about the face (information about skin colors, partssuch as eyes, a nose and a mouth), a method for constituting adiscriminator for the detection of the face by a learning algorithmtypified by a neural net, and the like. Also, in detecting the face,recognizing the face by combining the above methods in order to improvethe rate of the recognition is generally performed. For example, therehas been provided a method for detecting the face by utilizing theamount of a characteristic of the image and the wavelet transformation.

The template registering circuit (first registering unit) 202 registersthe partial image indicative of the subject as the target, as a templatethat is a model expressing the characteristics of the subject. Thetemplate matching circuit (first matching unit) 203 estimates the areaswith a high degree of similarity or a low degree of difference bycollating partial areas of the supplied images with the templateregistered by the template registering circuit 202, and altering thecollated partial area.

The histogram registering circuit (second registering unit) 204registers the histogram of the partial image indicative of the subjectas a target, as a model expressing the characteristics of the subject.The histogram matching circuit (second matching unit) 205 estimates theareas with a high degree of similarity or the a degree of difference bycollating histograms of the partial areas of the supplied images withthe histogram registered by the histogram registering circuit 204, andthen, altering the collated partial area.

The tracking process controlling circuit (tracking area determinationunit) 206 is configured by a CPU and the like, which controls thesubject tracking process. The subject detecting circuit 201 to thehistogram matching circuit 205 carry out the processing through thetracking process controlling circuit 206. The tracking processcontrolling circuit 206 determines the subject area from the evaluationvalues of the template matching circuit 203 and the histogram matchingcircuit 205. This determined subject area is set as output informationof the subject tracking circuit 109.

Next, a description will be given of a method configured to determinethe subject area. If there is the estimated area by the templatematching adjacent to the estimated area by the histogram matching, theestimated area by the template matching is adopted as the subject area.If there is no estimated area by the template matching adjacent to theestimated area by the histogram matching, the estimated area by thehistogram matching is adopted as the subject area. Also, the trackingprocess controlling circuit 206 performs the control of the partial areafor acquiring the amount of a characteristic and the control of theupdate timing for the template registering circuit 202 and the histogramregistering circuit 204. While a detailed description thereof will bedescribed below, briefly, the tracking with high precision is performedby controlling the partial area and the update timing that differs inaccordance with the property of the matching, in the templateregistering circuit 202 and the histogram registering circuit 204. In anembodiment of the present invention, although there is provided thecontrol circuit dedicated for the tracking process, the tracking processcontrolling circuit 206 may be configured so as to be included in thecontrol circuit 105.

Next, referring to FIG. 3A and FIG. 3B, a detailed description will begiven of the template matching. FIG. 3A is a diagram illustrating anexample of a subject model (template) in the template matching. Atemplate 301 is the partial image indicative of the subject set as atarget to be tracked (template), and sets the pixel pattern of thisimage as the amount of a characteristic. The amount of thecharacteristic 302 expresses the amount of the characteristic of eachcoordinate in a plurality of areas in the template 301, and in anembodiment of the present invention, the luminance signal of the pixeldata is set as the amount of a characteristic. The amount ofcharacteristic T(i, j) is expressed by a formula (1):[formula 1]T(i,j)={T(0,0),T(1,0), . . . ,T(W−1,H−1)}  (1)

where the coordinate in the template area is represented by (i, j), anda number in the horizontal direction is set represented by W, and anumber of pixels in the perpendicular direction is set represented by H.

FIG. 3B is a diagram indicative of the information about an image forsearching the subject as the target to be tracked. An image 303 is animage in a range in which the matching processing is performed. Thecoordinate in the searched image is represented by (x, y). A partialarea 304 is an area for acquiring an evaluation value for the matching.An amount of characteristic 305 expresses the amount of characteristicof the partial area 304, and the luminance signal of the image data isset as the amount of characteristic, as is the case with the template301. The amount of characteristic S(i, j) is expressed as a formula (2):[formula 2]S(i,j)={S(0,0),S(1,0), . . . ,S(W−1,H−1)}  (2)

wherein the coordinate within the partial area is represented by (i, j),the number of pixels in the horizontal direction is represented by W,and the number of pixels in the perpendicular direction is representedby H.

In an embodiment of the present invention, a value of SAD (Sum ofAbsolute Difference) is used as a calculating method configured toevaluate similarity between the template 301 and the partial area 304.The SAD value is calculated by formula (3):[formula 3]V(x,y)=Σ_(y=0) ^(H−1)Σ_(x=0) ^(W−1) |T(i,j)−S(i,j)|  (3)

The SAD value V(x, j) is calculated while shifting the partial area 304by one pixel in order from the upper left of the image 303 of thesearched range. The coordinate (x, y), in which the calculated V(x, j)is indicative of the minimum value, shows the position most similar tothe template 301. In other words, the position indicative of the minimumvalue is set as the position having a high possibility that the subjectas the target to be tracked is in the searched image.

Note that in an embodiment of the present invention, althoughone-dimensional information of the luminance signal is used as theamount of characteristic, three-dimensional information such as a signalfor brightness, hue, and saturation may be also used as the amount ofcharacteristic. Also, although a description has been given of the SADvalue as the calculating method of the evaluation value for thematching, the calculating method that is different therefrom, such asNormalized Cross Correlation, that is NCC (Normalized CorrelationCoefficient), may be used.

Next, referring to FIG. 4A and FIG. 4B, a detailed description will begiven of the histogram matching. FIG. 4A is a diagram illustrating anexample of a subject model in the histogram matching. A partial image401 is a partial image indicative of the subject set as the target to betacked, and the histogram generated from the pixel data of this partialimage 401 is set as the amount of characteristic. An amount ofcharacteristic 402 is expressed by a formula (4), if it is set as thehistogram for the M gradation of the luminance signal.[formula 4]p(m)={p(0),p(1), . . . p(M−1)}  (4)

FIG. 4B is a diagram illustrating information of an image for searchingthe subject as the target to be tracked. An image 403 is an image with arange in which the matching process is performed. The coordinate in thesearched image is expressed as (x, y). A partial area 404 is a partialarea for acquiring the evaluation value for the matching. An amount ofcharacteristic 405 is expressed as the amount of characteristicgenerated from the partial area 404, and the amount of characteristic405 is expressed by a formula (5) if it is set as the histogram for theM gradation of the luminance signal as is the case of the partial image401.[formula 5]q(m)={q(0),q(1), . . . q(M−1)}  (5)

The Bhattacharyya coefficient is used in a calculating method forestimating similarity between the histogram of the template 301 and thatof the partial area 404. The Bhattacharyya coefficient is calculated byformula (6):[formula 6]D(x,y)=Σ_(m=0) ^(M−1)√{square root over (p(m)×q(m))}  (6)

The Bhattacharyya coefficient D(x, y) is calculated by shifting thepartial area 404 by one pixel in order from the upper left of a searchedrange 403. The coordinate (x, y), in which the calculated D(x, y) isindicative of the maximum value, shows a position most similar to thepartial image 401. In other words, the position indicative of themaximum valve is the position having a high possibility that the subjectas the target to be tracked is in the searched image.

Here, although one-dimensional information of the luminance signal isdescribed as an example of the amount of characteristic,three-dimensional information such as a signal for brightness, hue, andsaturation may be also used as the amount of characteristic. Also,although a description is given of the Bhattacharyya coefficient, thecalculating method that is different therefrom, such as the histogramintersection, may be used.

In the image pickup apparatus 100 according to an embodiment of thepresent invention, the evaluation values and the estimated areas(estimated positions) by template matching and histogram matchingdetermines the subject area. The template matching uses a pattern of theimage data as the amount of characteristic. However, it is good at theclassification between similar subjects, but weak at changes inappearance. In contrast, the histogram matching does not utilize theimage data itself, but the histogram with respect to the amount ofcharacteristic of the matching. Since the histogram matching allows theamount of characteristic to remain ambiguous, although it is good at thechanges in appearance, such as the change of the attitude by thesubject, it is weak at the classification between the similar subjects.

In the method for tracking the subject, in addition to the method forexpressing the amount of characteristic, and that for the collating(matching) thereof as the above the formulae, how to acquire the amountof the characteristic becomes an important element. The method foracquiring the amount of characteristic is indicative of a magnitude ofan acquired area, and the update frequency of the amount ofcharacteristic. If the acquired area of the amount of characteristic islarger, the characteristic can be acquired for classifying between thesubject as the target to be tracked and other subjects, thereby allowingit to be good at the classification between similar subjects. At thesame time, it is likely to cause the change in a time direction to belarge, thereby causing it to be weak at changes in the appearance forthe subject as the target to be tracked. Since only the local area ofthe subject as the target to be tracked is set as the target if theacquired area of the amount of characteristic is smaller, the change inthe time direction is likely to be small, thereby allowing it to be goodat the changes in appearance of the subject as the target to be tracked.At this time, it is difficult to acquire the characteristics forclassifying between the subject as the target to be tracked and theother subjects, thereby causing it to be weak at the classificationbetween the similar subjects.

Also, with respect to the update frequency of the amount ofcharacteristic, if the update frequency is high, it causes a shortertime interval between the image at the timing of acquiring the amount ofcharacteristic and the image for the matching processing, therebyallowing the change of the subject as the target to be tracked to besmall. In other words, as a result, it becomes good at the changes inappearance in the subject as the target to be tracked. In contrast, forexample, if the amount of the characteristic is updated every trackingprocess, there is a concern that the estimation error for the subjecttracking may be included. Therefore, the reliability of the amount ofcharacteristic may be reduced. Thus, the reliability for the amount ofcharacteristic becomes high when the amount of characteristic is updatedonly if the reliability of the estimation for tracking the subject ishigh. However, the update frequency of the amount of characteristicbecomes reduced, and the time interval between the image at the timingfor acquiring the amount of characteristic and the image for thematching processing becomes large, thereby causing the large change ofthe subject as the target to be tracked. Therefore, it becomes weak atthe changes in appearance of the subject as the target to be tracked. Inthis case, the high reliability of the estimation for tracking thesubject is considered in the case for example, in which the subject asthe target to be tracked is detected by detecting the subject, and thelike.

Taking into account of the properties as described above, in thetemplate matching, the range in which the amount of characteristic isacquired is set small, and the update frequency of the amount ofcharacteristic is set high, thereby allowing a demerit of being poor atthe change in appearance in the template matching to be suppressed. Inthe histogram matching, the range in which the amount of characteristicis set large, and the update frequency of the amount of characteristicis set low, thereby improving the merit of being good at the changes inappearance in the histogram matching.

Next, FIG. 5 illustrates a framework for tracking the subject by thesubject tracking circuit 109. Images 501, 502, and 503 are examples ofinput images that are input by the subject tracking circuit 109. Theimage 501 is an image in which the subject as the target to be trackedis detected by the subject detecting circuit 201. The image 502 is animage at the time most proximate to the current time among the images towhich the subject tracking process has been performed. The image 503 isan image at the current time and an image on which process forestimating the subject area is performed. In a period from the image 501to the image 502, the subject as the target to be tracked is set so asnot to be detected by the subject detecting circuit 201.

Frames 505 and 506 show the estimated area that is estimated by thesubject tracking circuit 109. An area 504 is an area estimated by thesubject detecting circuit 201, and appears to have high reliability(degree of reliability) as the subject. Therefore, as shown in thepartial area 507, the amount of characteristic is acquired from therange larger than the area estimated by the subject detecting circuit201. A histogram 508 is generated from the partial area 507, and thishistogram 508 is registered by the histogram registering circuit 204. Atemplate 509 is the partial area indicative of the result immediatelybefore the subject tracking process and this partial area is registeredas the template by the template registering circuit 202. It is notedthat at the start of tracking the subject, the area estimated by thesubject detecting circuit 201 is registered as the template by thetemplate registering circuit 202.

A searched range 510 is a searched range in the subject trackingprocess. Histogram matching is carried out with the histogram 508 by thehistogram matching circuit 205 with respect to the searched range 510.In parallel with this, template matching is carried out by the templatematching circuit 203 using the template 509. Based on each matchingresult, a subject tracking area 506 is determined by the trackingprocess controlling circuit 206. In FIG. 5, it is shown that thehistogram 508, which is the amount of characteristic for the histogrammatching, has a larger range for acquiring the amount of characteristicand a lower update frequency than the template 509, which is the amountof characteristic for the template matching. In other words, the partialarea corresponding to the histogram registered by the histogramregistering circuit 204 is larger than the partial area corresponding tothe template registered by the template registering circuit 202. Also,the update frequency of the histogram registered by the histogramregistering circuit 204 is less than that of the template registered bythe template registering circuit 202.

Next, referring to FIG. 6, a description will be given of a subjecttracking process in an embodiment of the present invention. First, whenan image is supplied from the image processing circuit 106 to thesubject tracking circuit 109 (step S601), the subject tracking circuit109 detects the subject in order to determine the subject as a target tobe tracked (step S602). The detected subject area is set as the targetto be tracked, and then a histogram that is a subject model of thehistogram matching (amount of characteristic) is registered based on thepartial image indicative of the subject set as the target to be tracked(step S603). Also, the subject tracking circuit 109 registers thepartial image indicative of the subject set as the target to be tracked,as the template that is the subject model of the template matching(amount of characteristic) (step S604). In this case, the processing maybe stopped, if the subject is not detected, in step S602 (not shown).

Next, the image is supplied (input) from the image processing circuit106 to the subject tracking circuit 109 (step S605). Note that the inputimage in step S605 is indicative of the image in the area equivalent tothe searched image by the subject tracking circuit 109, and the inputimage in step S601 is different from that in step S605 with respect tothe time. Next, the detection of the subject is performed based on theinput image in step S605 (step S606). Subsequently, histogram matching(step S607) is performed and then, the template matching (step S608) isperformed. Following this, the subject area is determined based on theresult of the subject detection, the evaluation values of the histogrammatching, and the template matching, and the area information thereof(step S609).

Here, referring to FIG. 7, a description will be given of a flow ofsubject area determination processing in step S609. First, the subjecttracking circuit 109 determines whether or not the subject as the targetto be tracked is detected by the subject detection (step S701). If thesubject as the target to be tracked is detected in step S701 (YES), theestimated area by the subject detection is adopted as the subject area(step S702). On the other hand, if the subject as the target to betracked is not detected (NO), the step proceeds to step S703, and then,it is determined whether or not the reliability (degree of reliability)for the evaluation value of the histogram matching is high. In thedetermination for the reliability, for example, if the maximum value ofthe Bhattacharyya coefficient D(x, y) by the formula (6) is greater thanor equal to a predetermined threshold, the reliability is determined tobe high, and if the maximum value of the Bhattacharyya coefficient D(x,y) is less than the predetermined threshold, the reliability isdetermined to be low.

In step S703, if the reliability for the evaluation value of thehistogram matching is high (YES), the step proceeds to step S704, andthen it is determined whether or not there is the estimated area of thetemplate matching adjacent to the estimate area of the histogrammatching. This determination calculates a distance between thecoordinate (x, y) that is the maximum value of the Bhattacharyyacoefficient D(x, y) by the formula (6) and the coordinate (x, y) that isthe minimum value of the SAD value V(x, y) by the formula (3), and then,determines the proximity therebetween based on whether or not thecalculated distance is within the predetermined range. In step S704, ifthere is no estimated area of the template matching adjacent to theestimated distance of the histogram matching (NO), the step proceeds tostep S706, and then the estimated area by the histogram matching isadopted. In other words, the coordinate (x, y) that is the maximum valueof the Bhattacharyya coefficient D(x, y) by the formula (6) isdetermined as the position of the subject. On the other hand, if thereis the estimated area of the template matching adjacent to the estimateddistance of the histogram matching (YES), the step proceeds to stepS707, and then, the estimated area by the template matching is adopted.If there is the estimate area by the template matching adjacent to theestimated area of the histogram matching with the high reliability, itis contemplated that the estimated area by the template matching hasalso the high reliability. Also, since it is contemplated that thetemplate matching has higher precision for the position of the estimatedarea than the histogram matching, the estimated area of the templatematching is adopted in this case.

In contrast, in step S703, if the reliability for the evaluation valueof the histogram matching is low (NO), the step proceeds to step S705,and it is determined whether or not the reliability for the evaluationvalue of the template matching is high (step S705). The determinationfor the reliability determines the reliability to be high if the minimumvalue of the SAD value V(x, y) by the formula (3) is less than thepredetermined threshold, and determines the reliability to be low if theminimum value of the SAD value V(x, y) is greater than or equal to thepredetermined threshold.

In step S705, if the reliability for the evaluation value of thetemplate matching is high (Yes), the step proceeds to step S707. Thenthe estimated area by the template matching is adopted (step S707). Inother words, the coordinate (x, y) that is the minimum value of the SADvalue V(x, y) by the formula (3) is determined to be the position of thesubject. In contrast, if the reliability for the evaluation value of thetemplate matching is low in step S705 (NO), the step proceeds to stepS708 and neither of the estimation value of the template matching northe histogram matching is adopted.

Referring back to FIG. 6, when the subject area is determined in stepS609, the subject tracking circuit 109 performs the update processing ofthe subject model (amount of characteristic). Next, it is determinedwhether or not the reliability for the subject area determined in stepS610 is high. If the reliability is high in step S610 (YES), thehistogram is updated (step S611). In contrast, if the reliability is nothigh (low) (NO), the histogram is not updated. Note that as the methodconfigured to determine the reliability in step S610, the reliability isset high if the subject as the target to be tracked is detected by thesubject detection in step S606, while the reliability is set low if thesubject as the target to be tracked is not detected by the subjectdetection in step S606. In other words, the reliability may bedetermined to be high if the estimated area by the subject detection isadopted as the subject area by step S702, and it may be determined to below in the other cases.

Next, the subject tracking circuit 109 determines whether or not thetracking of the subject is continued (step S612). This determinationdetermines that the tracking is not continued if the estimated areas ofthe template matching and the histogram matching are not collectivelyadopted, as, for example, in step S708. On the other hand, the trackingis continued if the estimated area of either template matching or thehistogram matching is adopted as in step S706 and step S707. In stepS612, if the tracking is not continued (NO), the processing for trackingthe subject is completed. In other words, the processing is completed ifthe subject as the target to be tracked is not in the image of thesearched range. On the other hand, in step S612, if the tracking iscontinued (YES), the template is updated based on the estimated subjectarea (step S613). Then, back to step S605, the processing is repeatedlycarried out based on the sequentially supplied images. In this manner,the histogram used in the histogram matching is updated only when thereliability for the subject area is determined as high, even if thetracking has succeeded, while the template used in the template matchingis updated for every successful tracking.

Note that the template in the template matching and the acquired area ofthe histogram in the histogram matching are determined based on the areaas the result of the subject detection or the subject tracking. Also,although the magnitude of the template in the template matching isequivalent to the detected area that is detected by the subjectdetection, the magnitude of the acquired area of the histogram in thehistogram matching is larger than the area detected by the subjectdetection. If the magnitude of the acquired area of the histogram is setas the area that simply expands with respect to the area detected by thesubject detection, the acquired area may include the area that isdifferent from the subject as the target to be tracked, such as thebackground in the acquired area. In this case, since the precision ofthe subject tracking may be reduced, it is necessary to optimize theacquired area of the histogram.

Here, the optimized processing for the acquired area of the histogramdetermines the acquired area in accordance with the case of thedistribution for pixel information that is included in the detected areaby the subject detection and does not expand out of the area of thesubject. Here, referring to FIG. 8, a description will be given of aflow of the processing for optimizing the acquired area of the histogramby an optimizing unit (not shown). First, clustering is performed on theimage information for acquiring the histogram (step S801). This step isto make image information that is multi-dimensional intolower-dimensional image information by the clustering to allow thesuppression for the effect of noise and facilitate the handling of theinformation in the subsequent steps. As the method for the clustering,for example, general methods such as K-means may be applied.

Next, a labeling process is performed based on a clustering image (stepS802). In the labeling process, the same labeling is assigned to thesame class of the pixel information whose positions are consecutive (areadjacent vertically or horizontally). Next, with respect to a labelingimage, a label included in a first predetermined area is set as a validlabel, and a label not included in the first predetermined area is setas an invalid label (step S803). It is noted that the firstpredetermined area is determined based on the detected area of thesubject as the target to be tracked by the subject detection.

Next, with respect to the valid label, a pixel not included outside of asecond predetermined area different from the first predetermined area(that is, within the second predetermined area) is set as the validlabel, ant a pixel included outside of the second predetermined area(that is, out of the second predetermined area) is set as the invalidlabel (step S804). Note that in an embodiment of the present invention,the second predetermined area is set larger than the first predeterminedarea. Next, fitting is performed on the area based on the case of thedistribution for the valid label (step S805). Area fitting alters therectangular shape such that there are a lot of pixels that are the validlabels and a small number of pixels that are the invalid labels, as anexample.

Next, it is determined whether or not the estimated area satisfies apredetermined condition (step S806). If the estimated area satisfies thepredetermined condition (YES), the histogram is updated based on theestimated area (step S807). If the estimated area does not satisfy thepredetermined condition (NO), the histogram is not updated and theprocessing is stopped. For example, if the estimated area is smallerthan a predetermined threshold, or the estimated area is larger than thepredetermined threshold, the histogram is not updated. That is, even ifthe processing proceeds to step S611 as the reliability for the subjectarea is determined to be high in step S610, the histogram is not updatedif the estimated area does not satisfy the predetermined condition. Bythe above processing, the reduction in the precision of the tracking canbe prevented by optimizing the acquired area of the histogram.

As described above, the image processing apparatus according to anembodiment of the present invention can be provided such that theperformance for tracking the subject can be improved by optimizing theacquired range of the characteristics and the update frequency inaccordance with the properties of each matching, while combining thehistogram matching with the template matching.

Note that in an embodiment of the present invention, the subjecttracking apparatus is applied to the image pickup apparatus, however,the device applying the subject tracking apparatus is not limited to theimage pickup apparatus. For example, the subject tracking apparatus maybe applied to a display device configured to display the image (replaydata) supplied from an external device, the record medium, and the like.Note that in the display device, the subject tracking process isperformed as the replay data is set as the data of the subject trackingprocess. In this case, the control circuit such as the microcontrollerin the display device controls a displaying condition in displaying theimage, based on the information about the subject extracted by thesubject tracking process (the position, magnitude, and the like of thesubject in the image). More specifically, information representing thesubject, such as a frame, is displayed superimposed on the position ofthe subject in the image, or control of the luminance, color shade fordisplayed images and the like is exercised in accordance with theluminance or color information of the subject portion.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may be also referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2015-119267, filed Jun. 12, 2015, which is hereby incorporated byreference wherein in its entirety.

What is claimed is:
 1. A subject tracking apparatus that tracks asubject included in images that are sequentially supplied, the apparatuscomprising: one or more processors; and a memory storing instructionswhich, when the instructions are executed by the one or more processors,cause the subject tracking apparatus to function as units comprising: afirst registering unit configured to register a partial area indicativeof the subject in one image of the supplied images as a template; afirst matching unit configured to estimate a subject area by collating apartial area in a newly supplied image with the template registered bythe first registering unit; a second registering unit configured togenerate a histogram based on pixel values of a partial area indicativeof the subject in one image of the supplied images and register thegenerated histogram; a second matching unit configured to estimate asubject area by collating a histogram based on pixel values of a partialarea in a newly supplied image with the histogram registered by thesecond registering unit; and a tracking area determination unitconfigured to determine a tracking area based on either of a first areathat is the subject area estimated by the first matching unit or asecond area that is the subject area estimated by the second matchingunit, wherein the partial area of which the second registering unitgenerates the histogram based on the pixel values is larger than thepartial area registered as the template by the first registering unit.2. The subject tracking apparatus according to claim 1, wherein thesecond registering unit comprises an optimizing unit configured tooptimize a partial area for acquiring a histogram to be registered, andwherein the optimizing unit optimizes the partial area in accordancewith a condition of a distribution of pixel information included in thepartial area indicative of the subject and the pixel information beingnot expanding out of the area of the subject.
 3. The subject trackingapparatus according to claim 1, wherein the tracking area determinationunit determines either of a first area that is the subject areaestimated by the first matching unit or a second area that is thesubject area estimated by the second matching unit as the tracking area,in accordance with reliability for the collation by the first matchingunit and the second matching unit.
 4. The subject tracking apparatusaccording to claim 3, wherein the tracking area determination unitdetermines the second area as the tracking area if the reliability ofthe collation by the second matching unit is high.
 5. The subjecttracking apparatus according to claim 3, wherein the tracking areadetermination unit determines the first area as the tracking area if thereliability of the collation by the second matching unit is high and thefirst area is adjacent to the second area.
 6. The subject trackingapparatus according to claim 3, wherein the tracking area determinationunit determines the first area as the tracking area if the reliabilityof the collation by the second matching unit is low and the reliabilityof the collation by the first matching unit is high.
 7. The subjecttracking apparatus according to claim 1, wherein the second registeringunit generates the histogram based on the luminance of the pixelsincluded in the partial area indicative of the subject.
 8. The subjecttracking apparatus according to claim 1, wherein the second registeringunit generates the histogram based on the brightness, hue, andsaturation of the pixels included in the partial area indicative of thesubject.
 9. An image pickup apparatus comprising: an imaging elementconfigured to sequentially photoelectorically convert images of asubject to output captured images; one or more processors; and a memorystoring instructions which, when the instructions are executed by theone or more processors, cause the image pickup apparatus to function asunits comprising: a first registering unit configured to register apartial area indicative of the subject in one image of the suppliedimages as a template; a first matching unit configured to estimate asubject area by collating a partial area in a newly supplied image withthe template registered by the first registering unit; a secondregistering unit configured to generate a histogram based on pixelvalues of a partial area indicative of the subject in one image of thesupplied images and register the generated histogram; a second matchingunit configured to estimate a subject area by collating a histogrambased on pixel values of a partial area in a newly supplied image withthe histogram registered by the second registering unit; a tracking areadetermination unit configured to determine a tracking area based oneither of a first area that is the subject area estimated by the firstmatching unit or a second area that is the subject area estimated by thesecond matching unit; and a controlling unit configured to control animage capturing condition in image capture by the imaging element, inaccordance with information about the subject by the tracking areadetermination unit, wherein the partial area of which the secondregistering unit generates the histogram based on the pixel values islarger than the partial area registered as the template by the firstregistering unit.
 10. A control method for controlling a subjecttracking apparatus that tracks a subject included in images that aresequentially supplied, the method comprising: registering, by a firstregistering unit, a partial area indicative of the subject in one imageof the supplied images as a template; generating, by a secondregistering unit, a histogram based on pixel values of a partial areaindicative of the subject in one image of the supplied images andregistering the generated histogram; estimating, by a first matchingunit, a subject area by collating a partial area in a newly suppliedimage with the template registered by the first registering unit;estimating, by a second matching unit, a subject area by collating ahistogram based on pixel values of a partial area in a newly suppliedimage with the histogram registered by the second registering unit; anddetermining, by a tracking area determination unit, a tracking areabased on either of a first area that is the subject area estimated bythe first matching unit or a second area that is the subject areaestimated by the second matching unit, wherein the partial area of whichthe second registering unit generates the histogram based on the pixelvalues is larger than the partial area registered as the template by thefirst registering unit.
 11. An image processing apparatus that tracks asubject included in images that are sequentially input, the apparatuscomprising: one or more processors; and a memory storing instructionswhich, when the instructions are executed by the one or more processors,cause the image processing apparatus to function as units comprising: asubject detecting unit configured to detect a subject set as a target tobe tracked from the input images; a first registering unit configured toregister a partial area indicative of the detected subject set as thetarget to be tracked as a template; a first matching unit configured toestimate a first area with a high degree of similarity by collating eacharea of the partial area in the sequentially input images with thetemplate registered by the first registering unit; a second registeringunit configured to register a histogram of a partial area indicative ofthe detected subject set as the target to be tracked; a second matchingunit configured to estimate a second area with the high degree ofsimilarity by collating a histogram based on pixel values of a partialarea of the sequentially input images with the histogram registered bythe second registering unit; and a tracking area determination unitconfigured to determine a tracking area based on either of the firstarea or the second area, wherein the partial area corresponding to thehistogram registered by the second registering unit is the partial arealarger than the partial area corresponding to the template registered bythe first registering unit.
 12. The subject tracking apparatus accordingto claim 1, wherein the second matching unit collates each gradation ofthe histogram in the newly supplied image with that of the histogramregistered by the second registering unit to estimate the subject area.13. The subject tracking apparatus according to claim 1, wherein anupdate for the registration of the template by the first registeringunit is more frequent than that for the registration of the histogram bythe second registering unit.
 14. The image pickup apparatus according toclaim 9, wherein the second matching unit collates each gradation of thehistogram in the newly supplied image with that of the histogramregistered by the second registering unit to estimate the subject area.15. The image pickup apparatus according to claim 9, wherein an updatefor the registration of the template by the first registering unit ismore frequent than that for the registration of the histogram by thesecond registering unit.
 16. The control method according to claim 10,wherein, in the estimating by the second matching unit, each gradationof the histogram in the newly supplied image is collated with that ofthe histogram registered by the second registering unit to estimate thesubject area.
 17. The control method according to claim 10, wherein anupdate for the registration of the template by the first registeringunit is more frequent than that for the registration of the histogram bythe second registering unit.
 18. The image processing apparatusaccording to claim 11, wherein the second matching unit collates eachgradation of the histogram in the sequentially input images with that ofthe histogram registered by the second registering unit to estimate thesubject area.
 19. The image processing apparatus according to claim 11,wherein an update for the registration of the template by the firstregistering unit is more frequent than that for the registration of thehistogram by the second registering unit.